Evidence-based health policy in Germany: lack of communication and coordination between academia and health authorities?

Health-care decision making should consider the best available evidence, often in the form of systematic reviews (SRs). The number of existing SRs and their overlap make their identification and use difficult. Decision makers often rely on de novo SRs instead of using existing SRs. We describe two cases of duplicate reviews (minimum volume threshold of total knee arthroplasties and lung cancer screening) and one case of duplicate primary data analysis (transcatheter aortic valve implantation). All cases have in common that unintended duplication of research occurred between health authorities and academia, demonstrating a lack of communication and coordination between them.It is important to note that academia and health authorities have different incentives. Academics are often measured by the number of peer-reviewed publications and grants awarded. In contrast, health authorities must comply with laws and are commissioned to deliver a specific report within a defined period of time. Most replication is currently unintended. A solution may be the collaboration of stakeholders commonly referred to as integrated knowledge translation (IKT). The IKT approach means that research is conducted in collaboration with the end users of the research. It requires active collaborations between researchers and decision-makers or knowledge users (clinicians, managers, policy makers) throughout the research process. Wherever cooperation is possible in spite of requirements for independence or confidentiality, legal regulations should facilitate and support collaborative approaches between academia and health authorities.

[How robust are studies of currently permanently included digital health applications (DiGA)? Methodological quality of studies demonstrating positive health care effects of DiGA]. / Wie belastbar sind Studien der aktuell dauerhaft aufgenommenen digitalen Gesundheitsanwendungen (DiGA)? Methodische Qualität der Studien zum Nachweis positiver Versorgungseffekte von DiGA.

INTRODUCTION: Since September 2020 digital health applications (DiGA) can be prescribed by physicians and psychotherapists and are reimbursed within the German Statutory Health Insurance (SHI) system for the first time worldwide. For full reimbursement, the manufacturers have to provide evidence based on scientific studies that the DiGA can provide 'positive health care effects'. This study aims to analyze and evaluate the methodological quality of efficacy studies of DiGA in the categories 'Nervensystem' and 'Psyche' of the DiGA register that are permanently accepted. METHODS: The methodological quality was assessed using the revised Cochrane risk-of-bias tool for randomized trials (RoB 2). The risk of bias was assessed for the primary endpoint of each study according to an intention-to-treat analysis. RESULTS: Six DiGA were assessed for their methodological quality. Randomized controlled trials were conducted for all 6 DiGA that showed a high risk of bias, which was, in particular, due to a lack of blinding of the studies. In addition, drop-outs were significantly higher in the intervention group than in the control group in most studies. For most of the DiGA no published study protocol was available in advance so an analysis of a potential selective choice of the evaluation methodology was not possible. DISCUSSION: For reasons of transparency, verifiability, and comprehensibility of the study results, registration in a study registry and, more importantly, the publication of study protocols should be mandatory before the start of the studies. In addition, studies should be blinded by comparing the DiGA with a 'sham application' to reduce the high risk of bias. Differences in the drop-out rates of the investigated studies could indicate a lack of efficacy of the treatment in the intervention group, (technical) problems in the application of the DiGA, or a lack of motivation of the participants. CONCLUSION: The interim results 18 months after the introduction of DiGA in the German SHI system show that the studies on the evidence of the benefits of DiGA have a high potential for bias in certain areas. However, it should be positively emphasized that the manufacturers submitted randomized controlled trials to prove the medical benefit of the DiGAs investigated.

Disruptive technologies in health care disenchanted: a systematic review of concepts and examples.

OBJECTIVES: To clarify the concept of disruptive technologies in health care, provide examples and consider implications of potentially disruptive technologies for health technology assessment (HTA). METHODS: We conducted a systematic review of conceptual and empirical papers on healthcare technologies that are described as "disruptive." We searched MEDLINE and Embase from 2013 to April 2019 (updated in December 2021). Data extraction was done in duplicate by pairs of reviewers utilizing a data extraction form. A qualitative data analysis was undertaken based on an analytic framework for analysis of the concept and examples. Key arguments and a number of potential predictors of disruptive technologies were derived and implications for HTA organizations were discussed. RESULTS: Of 4,107 records, 28 were included in the review. Most of the papers included conceptual discussions and business models for disruptive technologies; only few papers presented empirical evidence. The majority of the evidence is related to the US healthcare system. Key arguments for describing a technology as disruptive include improvement of outcomes for patients, improved access to health care, reduction of costs and better affordability, shift in responsibilities between providers, and change in the organization of health care. A number of possible predictors for disruption were identified to distinguish these from "sustaining" innovations. CONCLUSIONS: Since truly disruptive technologies could radically change technology uptake and may modify provision of care patterns or treatment paths, they require a thorough evaluation of the consequences of using these technologies, including economic and organizational impact assessment and careful monitoring.

Around the Tables - Contextual Factors in Healthcare Coverage Decisions Across Western Europe.

BACKGROUND: Across Western Europe, procedures and formalised criteria for taking decisions on the coverage (inclusion in the benefits basket or equivalent) of healthcare technologies vary substantially. In the decision documents, which display the justification of, the rationale for, these decisions, national healthcare institutes may employ 'contextual factors,' defined here as situation-specific considerations. Little is known about how the use of such contextual factors compares across countries. We describe and compare contextual factors as used in coverage decisions generally and 4 decision documents specifically in Belgium, England, Germany, and the Netherlands. METHODS: Four group interviews with 3 experts from the national healthcare institute of each country, document and web site analysis, and a workshop with 1 to 2 of these experts per country were followed by the examination of the documents of 4 specific decisions taken in each of the 4 countries, sampled to vary widely in type of technology and decision outcome. RESULTS: From the available decision documents, we conclude that in every country studied, contextual factors are established 'around the table,' ie, in deliberation. All documents examined feature contextual factors, with similar contextual factor patterns leading to similar decisions in different countries. The Dutch decisions employ the widest variety of factors, with the exception of the societal functioning of the patient, which is relatively common in Belgium, England, and Germany. Half of the final decisions were taken in another setting, with the consequence that no documentation was retrievable for 2 decisions. CONCLUSION: First, we conclude that in these countries, contextual factors are actively integrated in the decision document, and that this is achieved in deliberation. Conceptualising contextual factors as both situation-specific and actively-integrated affords insight into practices of contextualisation and provides an encouragement for exchange between decision-makers on more qualitative aspects of decisions. Second, the decisions that lacked a publicly accessible justification of the final decision document raised questions on the decisions' legitimacy. Further research could address patterning of contextual factors, elucidate why some factors may remain implicit, and how decisions without a publicly available decision document may enable or restrain decision-making practice.

Testing a new taxonomic model for the assessment of medical devices: Is it plausible and applicable? Insights from HTA reports and interviews with HTA institutions in Europe.

OBJECTIVE: Medical devices (MDs) encompass a broad and heterogeneous range of technologies. While practices vary considerably across countries, MDs often find application in patient care with little or no evaluation of their effectiveness and safety following market approval. A recently proposed taxonomy of MDs considered devices from the viewpoint of Health Technology Assessment (HTA). The aim of the work presented here was to test its plausibility and applicability by considering real-world HTA practices. METHODS: HTA reports on MDs from European institutions were collected in a systematic manner and the evaluated devices and/or related procedures were matched to a position on the taxonomy. Following this, representatives from 16 European HTA institutions were asked about the usefulness of the taxonomy in semi-structured interviews. RESULTS: 1237 HTA reports (2004-2015) from 33 European institutions were included in the sample. The majority of reports was on technologies from the taxonomic positions initially estimated as having high relevance. Most of the experts interviewed stated that they found the taxonomy useful, particularly regarding its potential to aid in selecting technologies for assessment and to highlight potential methodological particularities per taxonomic position. CONCLUSIONS: Overall, the distribution of identified reports on the matrix confirmed that the initial estimation of the relevance and necessity of HTA provided in the taxonomic model is plausible. In addition, interviews with representatives of European HTA institutions showed that the taxonomy could be useful.

[GRADE Guidelines: 16. GRADE evidence to decision frameworks for tests in clinical practice and public health]. / GRADE-Leitlinien: 16. Von der Evidenz zur Empfehlung oder Entscheidung ­ Vorgehen nach GRADE für Tests in der klinischen Praxis und in Public Health.

OBJECTIVE: To describe the Grading of Recommendations Assessment, Development and Evaluation (GRADE) interactive Evidence to Decision (EtD) frameworks for tests and test strategies for clinical, public health or coverage decisions. STUDY DESIGN AND SETTING: As part of the GRADE Working Group's DECIDE project we conducted workshops, user testing with systematic review authors, guideline developers and other decision makers, and piloted versions of the EtD framework. RESULTS: EtD frameworks for tests share the structure, explicitness, and transparency of other EtD frameworks. They require specifying the purpose of the test, linked or related management and the key outcomes of concern for different test results and subsequent management. The EtD criteria address test accuracy and assessments of the certainty of the additional evidence necessary for decision-making. When there is no direct evidence of test effects on patient important outcomes, formal or informal modeling is needed to estimate effects. We describe the EtD criteria based on examples developed with GRADEpro (www.gradepro.org), GRADE's software that also provides interactive Summary of Findings Tables. CONCLUSION: EtD frameworks for developing recommendations and making decisions about tests lay out the sequential steps in reviewing and assessing the different types of evidence that need to be linked.

Scientific Evidence in Health Technology Assessment Reports: An In-Depth Analysis of European Assessments on High-Risk Medical Devices.

BACKGROUND: The aim of this study was to examine the scientific evidence on clinical effectiveness and safety used in health technology assessments (HTAs) of high-risk medical devices (MDs) in Europe. METHODS: We applied a systematic approach to identify European institutions involved in HTA and to select reports assessing MDs considered high-risk according to the definition in the new German health care regulation §137h. Reports published between 2010 and 2015 were considered in our subsequent analysis. We used a structured tool based on widely accepted methodologic principles from Drummond's framework to extract key information on the clinical evidence considered in the reports. RESULTS: Out of 1376 identified reports, 93 were eligible for analysis. All reports based their assessment primarily on direct evidence, in most cases (68%) identified through an independent systematic literature search. In more than half the identified studies considered in the reports, clinical evidence for demonstration of effectiveness and safety was of moderate or low quality. Even when systematic reviews and randomized controlled trials were available for assessment, most studies showed an unclear or high risk of bias. CONCLUSIONS: This study confirms that the quality of scientific evidence used in HTA of high-risk MDs is low and therefore the use of evidence needs improvement. The European Commission recently updated the regulation on MDs but mainly focused on the safety of materials and the CE (Conformité Européene [European Conformity]) mark. Our results show that additional changes are necessary, specifically with regard to the marketing authorization process of MDs, with stricter quality requirements based on methodologically robust trials, possibly in combination with other evidence sources.

Evidence-based decision-making for diagnostic and therapeutic methods: the changing landscape of assessment approaches in Germany.

This article examines the current status and most important changes over time to the legislative framework on the health technology assessment-informed decision-making process on diagnostic and therapeutic 'methods' in Germany. The relevant information was obtained through documentary analysis covering the period 1990 to 2017. The findings show that, even if the outpatient care sector appears to be much more regulated than the inpatient sector (based on a strict separation of the two care settings), developments in Germany have led to a more tightened assessment framework, making the use of evidence a firm component in the decision-making process. Nevertheless, a comprehensive approach for a systematic assessment of diagnostic and therapeutic 'methods' still does not exist. Readjustments of current regulations in Germany, such as the existing 'Verbotsvorbehalt' (i.e. provision of a diagnostic and therapeutic 'method' possible unless actively delisted) in the inpatient care setting, as well as further developments at the European level are needed in order to create a system that ensures early access to innovation under controlled study conditions.

QUALITY OF SAMPLE SIZE ESTIMATION IN TRIALS OF MEDICAL DEVICES: HIGH-RISK DEVICES FOR NEUROLOGICAL CONDITIONS AS EXAMPLE.

BACKGROUND: The aim of this study was to assess the quality of reporting sample size calculation and underlying design assumptions in pivotal trials of high-risk medical devices (MDs) for neurological conditions. METHODS: Systematic review of research protocols for publicly registered randomized controlled trials (RCTs). In the absence of a published protocol, principal investigators were contacted for additional data. To be included, trials had to investigate a high-risk MD, registered between 2005 and 2015, with indications stroke, headache disorders, and epilepsy as case samples within central nervous system diseases. Extraction of key methodological parameters for sample size calculation was performed independently and peer-reviewed. RESULTS: In a final sample of seventy-one eligible trials, we collected data from thirty-one trials. Eighteen protocols were obtained from the public domain or principal investigators. Data availability decreased during the extraction process, with almost all data available for stroke-related trials. Of the thirty-one trials with sample size information available, twenty-six reported a predefined calculation and underlying assumptions. Justification was given in twenty and evidence for parameter estimation in sixteen trials. Estimates were most often based on previous research, including RCTs and observational data. Observational data were predominantly represented by retrospective designs. Other references for parameter estimation indicated a lower level of evidence. CONCLUSIONS: Our systematic review of trials on high-risk MDs confirms previous research, which has documented deficiencies regarding data availability and a lack of reporting on sample size calculation. More effort is needed to ensure both relevant sources, that is, original research protocols, to be publicly available and reporting requirements to be standardized.

HTA of medical devices: Challenges and ideas for the future from a European perspective.

OBJECTIVES: Health Technology Assessment (HTA) of Medical devices (MDs) and MD-based procedures can be challenging due to the unique features and particularities of this group of technologies, such as device-operator interaction. The aim of this study was to (1) clarify, and supplement earlier findings on European HTA institutions' structural, procedural and methodological characteristics with regard to the assessment of MDs and to (2) capture the institutions' perceptions regarding challenges and future trends. METHODS: Semi-structured telephone interviews with 16 representatives from leading European HTA institutions were performed between April and July 2015. Summative and directed content analysis was used for the analysis, which is reported according to the COREQ checklist. RESULTS: Findings from the analysis of the interviews were manifold and partially confirmed what has been noted in the literature (e.g. scarce evidence; identifying relevant studies challenging due to more incremental innovations). Additional themes emerged that can be important for future considerations by HTA institutions and policy-makers alike (e.g. areas for future research; need for specific tools). CONCLUSIONS: The collective opinion of 16 European HTA institutions on MD evaluation could provide ideas to ameliorate the current regulatory situation beyond the modified EU regulation and start broader, more in-depth methodological discussions around the issue. Interviewed experts seem to agree that new approaches such as coverage with evidence development as some countries already introduced could help to overcome the problem with scarce evidence.

Methods for Health Economic Evaluation of Vaccines and Immunization Decision Frameworks: A Consensus Framework from a European Vaccine Economics Community.

BACKGROUND: Incremental cost-effectiveness and cost-utility analyses [health economic evaluations (HEEs)] of vaccines are routinely considered in decision making on immunization in various industrialized countries. While guidelines advocating more standardization of such HEEs (mainly for curative drugs) exist, several immunization-specific aspects (e.g. indirect effects or discounting approach) are still a subject of debate within the scientific community. OBJECTIVE: The objective of this study was to develop a consensus framework for HEEs of vaccines to support the development of national guidelines in Europe. METHODS: A systematic literature review was conducted to identify prevailing issues related to HEEs of vaccines. Furthermore, European experts in the field of health economics and immunization decision making were nominated and asked to select relevant aspects for discussion. Based on this, a workshop was held with these experts. Aspects on 'mathematical modelling', 'health economics' and 'decision making' were debated in group-work sessions (GWS) to formulate recommendations and/or--if applicable--to state 'pros' and 'contras'. RESULTS: A total of 13 different aspects were identified for modelling and HEE: model selection, time horizon of models, natural disease history, measures of vaccine-induced protection, duration of vaccine-induced protection, indirect effects apart from herd protection, target population, model calibration and validation, handling uncertainty, discounting, health-related quality of life, cost components, and perspectives. For decision making, there were four aspects regarding the purpose and the integration of HEEs of vaccines in decision making as well as the variation of parameters within uncertainty analyses and the reporting of results from HEEs. For each aspect, background information and an expert consensus were formulated. CONCLUSIONS: There was consensus that when HEEs are used to prioritize healthcare funding, this should be done in a consistent way across all interventions, including vaccines. However, proper evaluation of vaccines implies using tools that are not commonly used for therapeutic drugs. Due to the complexity of and uncertainties around vaccination, transparency in the documentation of HEEs and during subsequent decision making is essential.

TAXONOMY OF MEDICAL DEVICES IN THE LOGIC OF HEALTH TECHNOLOGY ASSESSMENT.

OBJECTIVES: The suitability of general HTA methodology for medical devices is gaining interest as a topic of scientific discourse. Given the broad range of medical devices, there might be differences between groups of devices that impact both the necessity and the methods of their assessment. Our aim is to develop a taxonomy that provides researchers and policy makers with an orientation tool on how to approach the assessment of different types of medical devices. METHODS: Several classifications for medical devices based on varying rationales for different regulatory and reporting purposes were analyzed in detail to develop a comprehensive taxonomic model. RESULTS: The taxonomy is based on relevant aspects of existing classification schemes incorporating elements of risk and functionality. Its 9 × 6 matrix distinguishes between the diagnostic or therapeutic nature of devices and considers whether the medical device is directly used by patients, constitutes part of a specific procedure, or can be used for a variety of procedures. We considered the relevance of different device categories in regard to HTA to be considerably variable, ranging from high to low. CONCLUSIONS: Existing medical device classifications cannot be used for HTA as they are based on different underlying logics. The developed taxonomy combines different device classification schemes used for different purposes. It aims at providing decision makers with a tool enabling them to consider device characteristics in detail across more than one dimension. The placement of device groups in the matrix can provide decision support on the necessity of conducting a full HTA.

[GRADE guidelines: 13. Preparing Summary of Findings tables and evidence profiles - continuous outcomes]. / GRADE Leitlinien: 13. Erstellen von Summary-of-Findings-Tabellen und Evidenzprofilen - kontinuierliche Endpunkte.

UNLABELLED: Presenting continuous outcomes in Summary of Findings tables presents particular challenges to interpretation. When each study uses the same outcome measure, and the units of that measure are intuitively interpretable (e.g., duration of hospitalisation, duration of symptoms), presenting differences in means is usually desirable. When the natural units of the outcome measure are not easily interpretable, choosing a threshold to create a binary outcome and presenting relative and absolute effects become a more attractive alternative. When studies use different measures of the same construct, calculating summary measures requires converting to the same units of measurement for each study. The longest standing and most widely used approach is to divide the difference in means in each study by its standard deviation and present pooled results in standard deviation units (standardised mean difference). Disadvantages of this approach include vulnerability to varying degrees of heterogeneity in the underlying populations and difficulties in interpretation. Alternatives include presenting results in the units of the most popular or interpretable measure, converting to dichotomous measures and presenting relative and absolute effects, presenting the ratio of the means of intervention and control groups, and presenting the results in minimally important difference units. We outline the merits and limitations of each alternative and provide guidance for meta-analysts and guideline developers. KEY POINTS: Summary of Findings tables provide succinct presentations of evidence quality and magnitude of effects. Summarising the findings of continuous outcomes presents special challenges to interpretation that become daunting when individual trials use different measures for the same construct. The most commonly used approach to providing pooled estimates for different measures, presenting results in standard deviation units, has limitations related to both statistical properties and interpretability. Potentially preferable alternatives include presenting results in the natural units of the most popular measure, transforming into a binary outcome and presenting relative and absolute effects, presenting the ratio of the means of intervention and control groups, and presenting results in preestablished minimally important difference units.

The new regulation to investigate potentially beneficial diagnostic and therapeutic methods in Germany: up to international standard?

Funding of diagnostic and therapeutic methods in Germany's statutory health insurance (SHI) follows a dichotomy: in outpatient care, only methods with proven benefit are reimbursed while in inpatient care, all methods may be provided unless they are excluded due to proven harm or lack of benefit. In January 2012, a new section 137e was added to the Social Code Book V (SGB V), allowing for the inclusion of innovative and potentially beneficial diagnostic or therapeutic methods in the SHI benefit basket, while additional evidence regarding their effectiveness and safety must be gathered. In 2013, the Federal Joint Committee (G-BA) has specified the details of this new approach, which can be considered a variety of "Coverage with Evidence Development" (CED). Our comparison with CED schemes in selected countries reveals a dependence of the CED implementation on the encompassing healthcare system. However, we identify a clear legislative foundation, a definitive decision-making body, the possibility to obtain public funding, and the preference for high quality study designs as constituting factors of an emerging international standard for CED. In addition, it is necessary to ensure the suitability of circumstances and technologies for the successful application of CED in a clear and transparent way.